Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Service Connector] az spring-cloud connection create postgres: Add --system-identity for springcloud-postgres connection #22459

Merged
merged 57 commits into from
Sep 19, 2022

Conversation

xfz11
Copy link
Member

@xfz11 xfz11 commented May 16, 2022

Related command

Description

Add --system-identity option for commands:
az spring-cloud/spring/webapp/containerapp connection create postgres/postgres-flexible/mysql-flexible/sql

It will enable the system identity of spring cloud app, and create an aad user in postgresql. Then create the connection.
Testing Guide

History Notes

[Component Name 1] BREAKING CHANGE: az command a: Make some customer-facing breaking change
[Component Name 2] az command b: Add some customer-facing feature


This checklist is used to make sure that common guidelines for a pull request are followed.

@xfz11 xfz11 changed the title [Azure/azure-cli] [Service Connector] az spring-cloud connection create postgres --system-identity: Add new auth type of springcloud-postgres connection [Service Connector] az spring-cloud connection create postgres --system-identity: Add new auth type of springcloud-postgres connection May 16, 2022
@xfz11 xfz11 changed the title [Service Connector] az spring-cloud connection create postgres --system-identity: Add new auth type of springcloud-postgres connection [Service Connector] --system-identity: Add new auth type of springcloud-postgres connection May 16, 2022
@yonzhan
Copy link
Collaborator

yonzhan commented May 16, 2022

Service Connector

@xfz11 xfz11 changed the title [Service Connector] --system-identity: Add new auth type of springcloud-postgres connection [Service Connector] az spring-cloud connection create: Add --system-identity for springcloud-postgres connection May 16, 2022
@xfz11 xfz11 changed the title [Service Connector] az spring-cloud connection create: Add --system-identity for springcloud-postgres connection [Service Connector] az spring-cloud connection create postgres: Add --system-identity for springcloud-postgres connection May 16, 2022
@xfz11
Copy link
Member Author

xfz11 commented May 16, 2022

@kairu-ms Could you help to check the build failure? What should I do to fix it?

@kairu-ms
Copy link
Contributor

/azp run

@azure-pipelines
Copy link

Azure Pipelines successfully started running 2 pipeline(s).

@xfz11
Copy link
Member Author

xfz11 commented May 17, 2022

@jiasli Hi Jiashuo, could you please help to check the "Build Rpm Package CentOS 7" pipeline error? What should I do to fix it?

@@ -139,3 +139,4 @@ vsts==0.1.25
wcwidth==0.1.7
websocket-client==1.3.1
xmltodict==0.12.0
psycopg2-binary==2.9.3
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@xfz11 do you add it in setup.py file of azure-cli

DEPENDENCIES = [
'antlr4-python3-runtime~=4.7.2',
'azure-appconfiguration~=1.1.1',
'azure-batch~=12.0.0',
'azure-cli-core=={}'.format(VERSION),
'azure-cosmos~=3.0,>=3.0.2',
'azure-data-tables==12.2.0',
'azure-datalake-store~=0.0.49',
'azure-graphrbac~=0.60.0',
'azure-keyvault-administration==4.0.0b3',
'azure-keyvault-keys==4.5.1',
'azure-keyvault~=1.1.0',
'azure-loganalytics~=0.1.0',
'azure-mgmt-advisor==9.0.0',
'azure-mgmt-apimanagement~=3.0.0',
'azure-mgmt-appconfiguration==2.1.0b2',
'azure-mgmt-applicationinsights~=1.0.0',
'azure-mgmt-authorization~=0.61.0',
'azure-mgmt-batchai==7.0.0b1',
'azure-mgmt-batch~=16.1.0',
'azure-mgmt-billing==6.0.0',
'azure-mgmt-botservice~=0.3.0',
'azure-mgmt-cdn==12.0.0',
'azure-mgmt-cognitiveservices~=13.1.0',
'azure-mgmt-compute~=26.1.0',
'azure-mgmt-consumption~=2.0',
'azure-mgmt-containerinstance~=9.1.0',
'azure-mgmt-containerregistry==8.2.0',
'azure-mgmt-containerservice~=19.1.0',
'azure-mgmt-cosmosdb==7.0.0b2',
'azure-mgmt-databoxedge~=1.0.0',
'azure-mgmt-datalake-analytics~=0.2.1',
'azure-mgmt-datalake-store~=0.5.0',
'azure-mgmt-datamigration~=10.0.0',
'azure-mgmt-deploymentmanager~=0.2.0',
'azure-mgmt-devtestlabs~=4.0',
'azure-mgmt-dns~=8.0.0',
'azure-mgmt-eventgrid==9.0.0',
'azure-mgmt-eventhub~=10.0.0',
'azure-mgmt-extendedlocation==1.0.0b2',
'azure-mgmt-hdinsight~=9.0.0',
'azure-mgmt-imagebuilder~=1.0.0',
'azure-mgmt-iotcentral~=9.0.0',
'azure-mgmt-iothub==2.2.0',
'azure-mgmt-iothubprovisioningservices==1.1.0',
'azure-mgmt-keyvault==9.3.0',
'azure-mgmt-kusto~=0.3.0',
'azure-mgmt-loganalytics==13.0.0b4',
'azure-mgmt-managedservices~=1.0',
'azure-mgmt-managementgroups~=1.0.0',
'azure-mgmt-maps~=2.0.0',
'azure-mgmt-marketplaceordering==1.1.0',
'azure-mgmt-media~=9.0',
'azure-mgmt-monitor~=3.0.0',
'azure-mgmt-msi~=6.0.0',
'azure-mgmt-netapp~=7.0.0',
'azure-mgmt-network~=19.3.0',
'azure-mgmt-policyinsights~=1.1.0b2',
'azure-mgmt-privatedns~=1.0.0',
'azure-mgmt-rdbms~=10.0.0',
'azure-mgmt-recoveryservicesbackup~=4.1.1',
'azure-mgmt-recoveryservices~=2.0.0',
'azure-mgmt-redhatopenshift==1.1.0',
'azure-mgmt-redis~=13.1.0',
'azure-mgmt-relay~=0.1.0',
'azure-mgmt-reservations==2.0.0', # TODO: Use requirements.txt instead of '==' #9781
'azure-mgmt-resource==21.1.0b1',
'azure-mgmt-search~=8.0',
'azure-mgmt-security==2.0.0b1',
'azure-mgmt-servicebus~=7.1.0',
'azure-mgmt-servicefabricmanagedclusters~=1.0.0',
'azure-mgmt-servicelinker==1.0.0',
'azure-mgmt-servicefabric~=1.0.0',
'azure-mgmt-signalr==1.0.0b2',
'azure-mgmt-sqlvirtualmachine==1.0.0b2',
'azure-mgmt-sql==4.0.0b1',
'azure-mgmt-storage~=20.0.0',
'azure-mgmt-synapse==2.1.0b2',
'azure-mgmt-trafficmanager~=1.0.0',
'azure-mgmt-web~=6.1.0',
'azure-multiapi-storage~=0.8.0',
'azure-storage-common~=1.4',
'azure-synapse-accesscontrol~=0.5.0',
'azure-synapse-artifacts~=0.12.0',
'azure-synapse-managedprivateendpoints~=0.3.0',
'azure-synapse-spark~=0.2.0',
'chardet~=3.0.4',
'colorama~=0.4.4',
# On Linux, the distribution (Ubuntu, Debian, etc) and version are checked for `az feedback`
'distro; sys_platform == "linux"',
'fabric~=2.4',
'javaproperties~=0.5.1',
'jsondiff~=1.3.0',
'packaging>=20.9,<22.0',
'PyGithub~=1.38',
'PyNaCl~=1.5.0',
'scp~=0.13.2',
'semver==2.13.0',
'six>=1.10.0', # six is still used by countless extensions
'sshtunnel~=0.1.4',
'urllib3[secure]',
'websocket-client~=1.3.1',
'xmltodict~=0.12'
]

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

BTW, I think it's the psycopg2-binary causes the failing of Build Rpm Package CentOS7.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why you need the binary library?

Copy link
Member Author

@xfz11 xfz11 May 18, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added in setup.py but build still failed.
If we don't use the psycopg2-binary, but install Psycopg from sources, python3-dev and libpq-dev package are required to build it. Any guide to add those dependencies in CLI build environment?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

https://dev.azure.com/azure-sdk/public/_build/results?buildId=1575771&view=logs&jobId=aedd744b-2bf4-59d5-bada-b121261b4f81&j=aedd744b-2bf4-59d5-bada-b121261b4f81&t=adba5b8c-e58b-5021-0563-de243d1b6648

2022-05-18T07:10:56.2535244Z Provides: azure-cli = 2.36.0-1.el7 azure-cli(x86-64) = 2.36.0-1.el7 config(azure-cli) = 2.36.0-1.el7 libcom_err-2abe824b.so.2.1()(64bit) libcrypto-2f2fcd35.so.1.1()(64bit) libcrypto.so.1.1(OPENSSL_1_1_0)(64bit) libcrypto.so.1.1(OPENSSL_1_1_0a)(64bit) libcrypto.so.1.1(OPENSSL_1_1_0c)(64bit) libcrypto.so.1.1(OPENSSL_1_1_0d)(64bit) libcrypto.so.1.1(OPENSSL_1_1_0f)(64bit) libcrypto.so.1.1(OPENSSL_1_1_0g)(64bit) libcrypto.so.1.1(OPENSSL_1_1_0h)(64bit) libcrypto.so.1.1(OPENSSL_1_1_0i)(64bit) libcrypto.so.1.1(OPENSSL_1_1_0j)(64bit) libcrypto.so.1.1(OPENSSL_1_1_1)(64bit) libcrypto.so.1.1(OPENSSL_1_1_1b)(64bit) libcrypto.so.1.1(OPENSSL_1_1_1c)(64bit) libcrypto.so.1.1(OPENSSL_1_1_1d)(64bit) libcrypto.so.1.1(OPENSSL_1_1_1e)(64bit) libcrypto.so.1.1(OPENSSL_1_1_1h)(64bit) libffi-c643fa1a.so.6.0.4()(64bit) libgssapi_krb5-497db0c6.so.2.2()(64bit) libgssapi_krb5.so.2(HIDDEN)(64bit) libgssapi_krb5.so.2(gssapi_krb5_2_MIT)(64bit) libk5crypto-b1f99d5c.so.3.1()(64bit) libk5crypto.so.3(HIDDEN)(64bit) libk5crypto.so.3(k5crypto_3_MIT)(64bit) libkeyutils-dfe70bd6.so.1.5()(64bit) libkeyutils.so.1(KEYUTILS_0.3)(64bit) libkeyutils.so.1(KEYUTILS_1.0)(64bit) libkeyutils.so.1(KEYUTILS_1.3)(64bit) libkeyutils.so.1(KEYUTILS_1.4)(64bit) libkeyutils.so.1(KEYUTILS_1.5)(64bit) libkrb5-fc820a1d.so.3.3()(64bit) libkrb5.so.3(HIDDEN)(64bit) libkrb5.so.3(krb5_3_MIT)(64bit) libkrb5support-a4e68a84.so.0.1()(64bit) libkrb5support.so.0(HIDDEN)(64bit) libkrb5support.so.0(krb5support_0_MIT)(64bit) liblber-2-f149c062.4.so.2.11.7()(64bit) libldap_r-2-aa060733.4.so.2.11.7()(64bit) libpcre-9513aab5.so.1.2.0()(64bit) libpq-30c1c89d.so.5.14()(64bit) libsasl2-8fd1a564.so.3.0.0()(64bit) libselinux-0922c95c.so.1()(64bit) libssl-6b7b94d5.so.1.1()(64bit) libssl.so.1.1(OPENSSL_1_1_0)(64bit) libssl.so.1.1(OPENSSL_1_1_0d)(64bit) libssl.so.1.1(OPENSSL_1_1_1)(64bit) libssl.so.1.1(OPENSSL_1_1_1a)(64bit)

2022-05-18T07:10:56.2544788Z Requires(rpmlib): rpmlib(CompressedFileNames) <= 3.0.4-1 rpmlib(FileDigests) <= 4.6.0-1 rpmlib(PayloadFilesHavePrefix) <= 4.0-1

2022-05-18T07:10:56.2551092Z Requires: ld-linux-x86-64.so.2()(64bit) ld-linux-x86-64.so.2(GLIBC_2.3)(64bit) libc.so.6()(64bit) libc.so.6(GLIBC_2.12)(64bit) libc.so.6(GLIBC_2.14)(64bit) libc.so.6(GLIBC_2.16)(64bit) libc.so.6(GLIBC_2.17)(64bit) libc.so.6(GLIBC_2.2.5)(64bit) libc.so.6(GLIBC_2.3)(64bit) libc.so.6(GLIBC_2.3.2)(64bit) libc.so.6(GLIBC_2.3.4)(64bit) libc.so.6(GLIBC_2.4)(64bit) libc.so.6(GLIBC_2.6)(64bit) libc.so.6(GLIBC_2.7)(64bit) libc.so.6(GLIBC_2.8)(64bit) libcom_err-2abe824b.so.2.1()(64bit) libcrypto-2f2fcd35.so.1.1()(64bit) libcrypto-2f2fcd35.so.1.1(OPENSSL_1_1_0)(64bit) libcrypto-2f2fcd35.so.1.1(OPENSSL_1_1_0d)(64bit) libcrypto-2f2fcd35.so.1.1(OPENSSL_1_1_0f)(64bit) libcrypto-2f2fcd35.so.1.1(OPENSSL_1_1_0i)(64bit) libcrypto-2f2fcd35.so.1.1(OPENSSL_1_1_1)(64bit) libdl.so.2()(64bit) libdl.so.2(GLIBC_2.2.5)(64bit) libffi-c643fa1a.so.6.0.4()(64bit) libgssapi_krb5-497db0c6.so.2.2()(64bit) libgssapi_krb5-497db0c6.so.2.2(gssapi_krb5_2_MIT)(64bit) libk5crypto-b1f99d5c.so.3.1()(64bit) libk5crypto-b1f99d5c.so.3.1(k5crypto_3_MIT)(64bit) libkeyutils-dfe70bd6.so.1.5()(64bit) libkeyutils-dfe70bd6.so.1.5(KEYUTILS_0.3)(64bit) libkeyutils-dfe70bd6.so.1.5(KEYUTILS_1.0)(64bit) libkeyutils-dfe70bd6.so.1.5(KEYUTILS_1.5)(64bit) libkrb5-fc820a1d.so.3.3()(64bit) libkrb5-fc820a1d.so.3.3(krb5_3_MIT)(64bit) libkrb5support-a4e68a84.so.0.1()(64bit) libkrb5support-a4e68a84.so.0.1(krb5support_0_MIT)(64bit) liblber-2-f149c062.4.so.2.11.7()(64bit) libldap_r-2-aa060733.4.so.2.11.7()(64bit) libpcre-9513aab5.so.1.2.0()(64bit) libpthread.so.0()(64bit) libpthread.so.0(GLIBC_2.2.5)(64bit) libpthread.so.0(GLIBC_2.3.2)(64bit) libresolv.so.2()(64bit) libresolv.so.2(GLIBC_2.2.5)(64bit) libsasl2-8fd1a564.so.3.0.0()(64bit) libselinux-0922c95c.so.1()(64bit) libssl-6b7b94d5.so.1.1()(64bit) libssl-6b7b94d5.so.1.1(OPENSSL_1_1_0)(64bit) libssl-6b7b94d5.so.1.1(OPENSSL_1_1_1)(64bit) libz.so.1()(64bit) rtld(GNU_HASH)
...
Step 13/13 : RUN rpm -i ./azure-cli-dev.rpm &&     az --version
 ---> Running in db374f835afc
error: Failed dependencies:
	libcrypto-2f2fcd35.so.1.1(OPENSSL_1_1_0)(64bit) is needed by azure-cli-2.36.0-1.el7.x86_64
	libcrypto-2f2fcd35.so.1.1(OPENSSL_1_1_0d)(64bit) is needed by azure-cli-2.36.0-1.el7.x86_64
	libcrypto-2f2fcd35.so.1.1(OPENSSL_1_1_0f)(64bit) is needed by azure-cli-2.36.0-1.el7.x86_64
	libcrypto-2f2fcd35.so.1.1(OPENSSL_1_1_0i)(64bit) is needed by azure-cli-2.36.0-1.el7.x86_64
	libcrypto-2f2fcd35.so.1.1(OPENSSL_1_1_1)(64bit) is needed by azure-cli-2.36.0-1.el7.x86_64
	libgssapi_krb5-497db0c6.so.2.2(gssapi_krb5_2_MIT)(64bit) is needed by azure-cli-2.36.0-1.el7.x86_64
	libk5crypto-b1f99d5c.so.3.1(k5crypto_3_MIT)(64bit) is needed by azure-cli-2.36.0-1.el7.x86_64
	libkeyutils-dfe70bd6.so.1.5(KEYUTILS_0.3)(64bit) is needed by azure-cli-2.36.0-1.el7.x86_64
	libkeyutils-dfe70bd6.so.1.5(KEYUTILS_1.0)(64bit) is needed by azure-cli-2.36.0-1.el7.x86_64
	libkeyutils-dfe70bd6.so.1.5(KEYUTILS_1.5)(64bit) is needed by azure-cli-2.36.0-1.el7.x86_64
	libkrb5-fc820a1d.so.3.3(krb5_3_MIT)(64bit) is needed by azure-cli-2.36.0-1.el7.x86_64
	libkrb5support-a4e68a84.so.0.1(krb5support_0_MIT)(64bit) is needed by azure-cli-2.36.0-1.el7.x86_64
	libssl-6b7b94d5.so.1.1(OPENSSL_1_1_0)(64bit) is needed by azure-cli-2.36.0-1.el7.x86_64
	libssl-6b7b94d5.so.1.1(OPENSSL_1_1_1)(64bit) is needed by azure-cli-2.36.0-1.el7.x86_64
The command '/bin/sh -c rpm -i ./azure-cli-dev.rpm &&     az --version' returned a non-zero code: 1

I think there is some problem with psycopg2-binary that it requires libs like libcrypto-2f2fcd35.so.1.1(OPENSSL_1_1_0)(64bit), but doesn't provide it, meaning it is not compatible with rpmbuild and rpm. This could be similar to #20608.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We should use psycopg2 as dependency when build package, which is suggested by psycopg owner. It has some pre-required package, to reduce the complexity, we may use psycopg2-binary in develop environment.

Copy link
Member Author

@xfz11 xfz11 Jun 13, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We should use psycopg2 as dependency when build package, which is suggested by psycopg owner. It has some pre-required package, to reduce the complexity, we may use psycopg2-binary in develop environment.

Thanks for your help! Since psycopg2 has other library dependencies on Linux and MacOS. I update the code to dynamically install the packages only when user runs the command the first time. For windows, it needs no other library, so I add the package in windows requirement files to improve user experience.

@@ -152,7 +152,8 @@
'sshtunnel~=0.1.4',
'urllib3[secure]',
'websocket-client~=1.3.1',
'xmltodict~=0.12'
'xmltodict~=0.12',
'psycopg2-binary==2.9.3'
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please add dependencies alphabetically.

@yonzhan yonzhan removed this from the May 2022 (2022-05-24) - For Build milestone May 18, 2022
Copy link
Contributor

@houk-ms houk-ms left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can you leave more comments on the _creadential_free implementation>?

oauthlib==3.0.1
packaging==21.3
paramiko==2.10.1
pbr==5.3.1
pkginfo==1.8.2
portalocker==2.3.2
psutil==5.9.0
psycopg2==2.9.3
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why is it required for windows only?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

psycopg2 has no pre-compiled binary version on Macos and some linux distribution. It needs to compile from source code and needs some extra library installed. In order not to impact too many CLI users, we will install packages dynamically when users run the specific command.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

psycopg2 currently cannot be installed on Python 3.11 (psycopg/psycopg2#1514), which blocks Azure CLI's Python 3.11 support (#24494).

@xfz11
Copy link
Member Author

xfz11 commented Sep 19, 2022

@kairu-ms Could you please help review the PR?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

7 participants